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Abstract 

The  integration  of  vision  sensors  and  inertial  navigation  system  (INS)  can  enable  precision 
navigation  capabilities  in  the  absence  of  GPS.  Inspired  from  biological  systems,  a  multi-aperture 
vision  processing  system  allows  for  accurate  self-motion  (egomotion)  estimation  by  observing 
optical  flow  across  all  apertures.  The  multi-aperture  approach  is  particularly  well  suited  for 
resolving  motion- ambiguity  by  providing  a  wide  field  of  regard  for  detecting  and  tracking  visual 
features  (optical  flow).  This  paper  presents  a  data  fusion  approach  for  multi-aperture  sensors  that 
integrates  the  vision  processing  into  a  single  unified  frame  of  reference  by  projecting  imagery 
from  each  aperture  onto  the  unit  sphere  centered  on  the  navigation  frame.  The  unit  sphere 
projection  allows  for  the  seamless  integration  of  multiple  apertures  into  the  more  natural  angle- 
angle  space  of  the  navigation  frame  of  reference. 

As  a  first  step  in  evaluating  the  multi-aperture  processing  strategy,  algorithms  are  evaluated 
based  on  simulated  data.  The  results  presented  clearly  show  the  advantage  of  coupling  the 
inertial  system  with  a  multi- aperture  optical  system. 
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Introduction 

Vision-based  navigation  techniques  serve  as  a  viable  option  for  autonomous,  passive  navigation 
and  guidance  in  Global  Navigation  Satellite  Systems  (GNSS)-denied  environments  [1].  This 
paper  discusses  extension  of  the  vision-aided  inertial  navigation  approach  for  multi- aperture 
camera  cases.  Inspired  from  biological  systems,  a  multi-aperture  vision  processing  system  allows 
for  accurate  motion  estimation  by  observing  optical  flow  across  all  apertures.  The  multi-aperture 
approach  is  particularly  well  suited  for  resolving  motion  ambiguity  by  providing  a  wide  filed  of 
regard  for  detecting  and  tracking  visual  features.  Moreover,  as  we  show  in  this  paper,  a  multi¬ 
aperture  system  coupled  with  an  inertial  system  simplifies  the  problem  of  obtaining  the  range  to 
the  observed  features  as  our  system  moves  though  the  environment. 

The  multi-aperture  vision  formulation  presented  herein  is  inspired  by  a  wide  field  of  regard  of 
insect  vision  sensors  such  as  compound  eyes  of  a  dragonfly  shown  in  Figure  1 . 


Figure  1.  Example  biological  multi-aperture  vision  system:  compound  eyes  of  a  dragon  fly 

Insects  navigate  successfully  and  efficiently  in  complex  environments  using  optical  flow  sensors 
coupled  with  other  sensors  such  as  halteres  (inertial  sensors).  Rather  that  using  a  single  high- 
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quality  vision  device,  an  eye  of  an  insect  can  be  represented  as  a  combination  of  multiple  low- 
quality  cameras  with  slightly  different  fields-of-view  and  optical  characteristics.  Moreover,  the 
coupling  of  vision  with  other  sensors  provides  more  robust  egostate  estimation  than  by  using 
vision  alone.  Therefore,  this  paper  fuses  multi-aperture  vision  sensing  with  inertial  navigation. 

Furthermore,  the  use  of  multi- aperture  vision  is  also  motivated  by  the  enhancement  of  situational 
awareness  for  guidance  and  mission  planning:  the  capability  to  look  forward  is  augmented  by 
side-looking  and  backward-looking  capabilities.  Figure  2  illustrates  the  improved  situational 
awareness  that  is  achieved  by  using  multi-aperture  vision  as  compared  to  a  single  aperture 
camera  system  for  an  indoor  flight  mission  of  a  mini-UAV. 

Single-aperture  view 

Optimal 
motion  path 


Multi-aperture  view 


Optimal 
motion  path 


Figure  2.  Single-aperture  vs.  multi-aperture  example 

In  this  example,  the  limited  field-of-view  of  a  single  aperture  results  in  the  mission  path  that 
collides  into  the  window.  The  multi-aperture  camera  allows  for  a  complete  observation  of  the 
scene  and  allows  avoiding  the  collision. 
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Figure  3  illustrates  an  example  implementation  of  the  multi-aperture  camera  system. 


Figure  3,  Multi-aperture  experimental  setup  developed  by  the  Alt-Nav  team  of  the  Air  Force  Research 
Laboratory’s  Munitions  Directorate;  the  setup  includes  three  video  cameras  with  a  90-deg  separation  of  their 

optical  axes 

This  paper  focuses  on  the  benefits  of  using  multi- aperture  system  for  the  estimation  of  vehicle 
navigation  states  including  position,  velocity,  and  attitude.  The  main  benefits  are  summarized  as 
follows: 

•  Increased  number  of  high  quality  features  that  can  be  applied  for  navigation; 

•  Improved  relative  feature  geometry,  which  results  in  reduced  values  of  dilution  of  precision 
(DOP)  factors  for  the  estimation  of  navigation  parameters  from  vision  features;  and, 

•  Improved  capability  to  resolve  the  unknown  scale  of  video  images. 

The  remainder  of  the  paper  is  organized  as  follows.  First,  a  unit  sphere  coordinate  frame  that  is 
applied  to  represent  multi-aperture  imagery  is  described.  Second,  motion  constraints  are  derived 
in  the  unit  sphere  frame.  Third,  fusion  of  multi-aperture  vision  data  with  measurements  of  an 
inertial  navigation  system  (INS)  is  discussed.  Finally,  simulation  results  are  presented  to 
demonstrate  efficiency  of  the  multi-aperture  vision  approach. 
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Unit  Sphere  Representation  of  Multi -Aperture  Imagery 


A  unified  coordinate  frame  has  to  be  chosen  for  multi- aperture  image  processing.  Particularly, 
features  observed  by  different  cameras  must  be  converted  into  a  single  frame  of  reference.  This 
paper  assumes  that  point  features  are  extracted  from  video  images  and  applied  for  navigation 
purposes.  Considerations  herein  are  readily  extendable  for  other  types  of  features  such  as  line 
features  and  planar  surfaces. 

Each  camera  originally  represents  its  pixels  by  their  two-dimensional  (2D)  Cartesian  coordinates 
resolved  in  the  camera  focal  plane  (these  coordinates  are  generally  referred  to  as  homogeneous 
coordinates  [2]).  In  principle,  a  Cartesian  representation  can  be  also  adapted  for  the  unified 
frame,  which,  in  this  case,  will  be  represented  by  a  planar  surface.  However,  this  representation 
has  a  drawback  of  singularity  cases  as  shown  in  Figure  4. 


Unified  plane 


Feature  unit 


No  projection  exists  for 
features  with  unit  vectors  e 
coUinear  to  the  unified 
planar  surface. 


Unified  plane 

For  features  whose  unit 
vectors  are  almost 
coUinear  to  the  unified 
plane,  small  feature  j 

errors  result  in  large  / 
projection  errors. 


Focal  plane 


Focal  point 


Figure  4.  Drawbacks  of  the  Cartesian  representation  of  the  unified  coordinate  frame  for  the  multi-aperture 


camera  case 
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Homogeneous  feature  coordinates  extracted  from  an  image  of  a  particular  camera  must  be 
projected  onto  a  planar  surface  that  is  chosen  for  the  multi-aperture  frame.  For  those  features 
whose  unit  vector  (i.e.,  a  unit  vector  pointed  from  the  camera  focal  point  to  the  feature)  is 
collinear  to  the  unified  plane,  such  a  projection  does  not  exist.  If  the  unit  vector  is  almost 
collinear  to  the  unified  plane,  then  small  errors  in  feature  measurements  result  in  large  projection 
errors. 

To  remove  singular  cases  associated  with  the  Cartesian  representation,  a  unit  sphere  approach  is 
chosen  for  multi-aperture  cameras.  This  approach  is  motivated  by  nature’s  multi-aperture  vision 
systems,  such  as  the  compound  eyes  of  insects,  which  process  optical  flow  by  projecting  it  onto  a 
sphere.  The  unit  sphere  provides  a  natural  framework  for  wide  field-of-view  (>180  degree) 
sensors,  where  the  traditional  focal  plane  representation  is  not  possible.  Figure  5  shows  the  unit 
sphere  feature  projection. 


Figure  5.  Unit  sphere  frame  and  unit  sphere  feature  projection 
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Features  that  are  extracted  from  multiple  cameras  are  projected  on  the  surface  of  a  unit  sphere 
and  represented  by  their  spherical  azimuth  and  elevation  angles.  In  Figure  5,  the  unit  sphere 
origin  is  collocated  with  the  camera’s  focal  point.  However,  this  is  not  required,  and  the  unit 
sphere  projection  can  be  generalized  for  cases  where  cameras’  focal  points  are  not  collocated 
with  the  sphere  origin  and  with  each  other.  Figure  6  illustrates  the  feature  projection  for  the  non- 
collocated  case. 


Figure  6.  Unit  sphere  feature  projection  for  the  case  of  non-collocated  cameras 
Equation  (1)  defines  unit  sphere  projections: 

Rf=cr-K  Pf-Rc 

ef=Rf/||Rf||  (1) 

(p  =  arctan(ejy  ,eg; )  (p  =  arcsiniCf^) 

where: 

is  the  coordinate  transformation  matrix  from  the  camera  {Xc,Yc,Zc} -frame  into  the 
{Xs,Ys,Zs} -frame  attached  to  the  unit  sphere;  m,j,fhy  are  homogeneous  feature  coordinates 
extracted  from  the  image; 

F  is  the  camera  focal  length; 


7 


Rj,  is  the  vector  that  originates  from  the  unit  sphere  origin  and  ends  at  the  camera  focal 


point;  and, 


arctan(,)  is  the  four-quadrant  arctangent  function. 

The  main  advantage  of  the  unit  sphere  formulation  is  the  removal  of  singularity  projection  cases 
that  are  present  if  Cartesian  representation  is  utilized  (see  Figure  4  above). 

Formulation  of  Motion  Constraint  Equations  in  the  Unit  Sphere  Frame 

The  use  of  the  unit  sphere  feature  projections  requires  the  transformation  of  motion  equations 
(i.e.,  relationships  between  feature  parameters  and  motion  parameters)  from  the  Cartesian  frame 
into  the  spherical  frame.  This  transformation  is  illustrated  below  for  a  2D  motion  case  where 
only  a  translation  motion  component  is  present.  In  this  case,  the  unified  frame  is  represented  by  a 
unit  circle  as  shown  in  Figure  7. 


Feature  at  image  1 


Figure  7.  Unit  circle  representation  for  a  2D,  translation  only  motion  case 
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From  the  geometry  shown  in  Figure  3: 


a/cos  A(p  =  b/sinA(p  (2) 

where: 

a=pW-(AR,e«); 
b  =  -(AR,e^'»); 

A(p  =  ; 

(,)  is  the  vector  dot  product; 

=  ||^***||  distance  to  the  feature  for  image  1; 

is  the  feature  unit  vector  for  image  1;  and,  is  the  unit  vector  perpendicular  to  e*'* . 
Correspondingly,  the  following  constraint  equation  is  formulated: 

—  (AR,e^'^))-sinA(p  =  — (AR,e®  )-cos  Acp  (3) 

This  equation  can  be  rearranged  as  follows: 

-  sinAcp  =  — (AR,e^^)  (4) 

Note  that  Acp  is  directly  calculated  from  camera  measurements.  Also,  equation  (4)  only  includes 
the  unknown  range  to  the  feature  at  the  first  image.  The  knowledge  of  range  information  for 
other  images  (second,  third,  etc.)  is  not  required.  Once  the  initial  range  is  estimated,  the  motion 
constraint  (4)  can  be  applied  to  formulate  Kalman  filter  vision/inertial  measurement  observables 
for  other  images  without  the  need  to  know  (or  measure)  their  feature  ranges.  As  compared  to  a 
complete  motion  formulation  that  includes  feature  range  variables  for  the  current  image,  the 
motion  constraint  formulation  is  beneficial  for  a  monocular  camera  case,  as  well  as  for  stereo 
cases  with  a  limited  baseline,  where  direct  measurements  of  the  range  value  are  not  possible  or 
inefficient. 
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The  unit  sphere  formulation  is  particularly  well  suited  for  representing  cases  of  the  rotational 


motion  only.  Figure  8  exemplifies  the  rotational  motion  case. 


Figure  8.  3D  angular  motion  represented  in  the  unit  sphere  frame 


In  this  case,  camera  rotation  directly  translates  into  the  rotation  of  the  feature  unit  vector.  This  is 
expressed  as  follows: 


(5) 


For  a  general  3D  case  that  includes  translational  and  rotational  motion  components,  two  motion 
constraint  equations  are  derived  in  a  manner  similar  to  the  2D  consideration  above.  The  first 
constraint  is  obtained  by  considering  the  horizontal  component  of  the  translational  motion,  and 
the  second  constraint  is  formulated  by  adding  the  vertical  motion  to  it.  Constraint  equations  are 
then  modified  to  incorporate  the  rotational  motion  component.  The  final  constraint  equations  are 
as  follows: 


(6) 


where  AR  and  AC^  are  position  and  orientation  changes  between  images  1  and  2;  and,  matrices 
B  and  D  are  defined  as: 
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0-10 


B=  1  0  0 

0  0  0 


D=  0 


cos({p^*^)  sin({p*^*^) 


0 


0  -cos({p^'*) 

0  -sin({p*^*') 

0 


Multi-Aperture/Internal  Data  Fusion 

Multi- aperture/INS  data  fusion  is  formulated  in  the  feature  domain  using  the  complementary 
Kalman  filter  methodology  [3].  In  this  case,  the  multi-aperture  vision/inertial  Kalman  filter 
estimates  inertial  error  states.  A  fifteen-state  INS  error  model  is  used  (including  position  errors, 
velocity  errors,  attitude  errors,  gyro  drifts  and  accelerometer  biases)  and  INS  error  propagation 
equations  [4]  are  applied  to  represent  the  dynamic  state  model.  This  section  formulates  the  filter 
measurement  observables. 

Motion  constraint  equations  (6)  are  applied  to  formulate  Kalman  filter  observables  rip  for  the 
multi-aperture  vision  inertial  aiding: 


■ACVD-ARBjs-fe"’. 


(8) 


P  =  i,...p 


where: 

e^"\p  =  l,..,P,s  =  l,2  is  the  unit  vector  of  the  feature  p  that  is  extracted  from  image  s;  is  the 
estimated  feature  range  for  image  1,  and,  ARj^s  and  AC^  are  INS  position  and  orientation 


changes  between  images.  Image  I  is  generally  the  image  where  the  feature  was  first  observed 


and  image  n  is  the  current  image. 
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Equation  (8)  is  linearized  to  support  the  linear  formulation  of  the  complementary  Kalman  filter: 


p,AR' 


+  H 


5RiNs+Hp„5\|/ 


INS 


P.f 


p 

50 

p  =  l,...,P 


(9) 


where  are  SRj^s  and  5\|/,ns  are  INS  position  and  attitude  errors;  are  feature  state 

errors  (including  initial  angular  errors  and  initial  range  errors);  and,  5(pp"\59p^  are  vision 
measurement  errors  for  the  current  update  epoch.  The  filter  state  vector  includes  both  inertial 
error  states  and  feature  error  states.  As  mentioned  previously,  the  inertial  error  state  vector  is  a 
fifteen-state  vector  comprised  of  position  error  states  (three  states),  velocity  error  states  (three 
states),  attitude  error  states  (three  states),  gyro  drift  states  (three  states),  and  accelerometer  bias 
states  (three  states).  Incorporation  of  vision  feature  error  states  into  the  Kalman  filter  state  vector 
is  similar  to  simultaneous  localization  and  mapping  (SLAM)  formulations  (see,  for  example, 
[5]). 


The  initial  feature  range 


p*'*  is  estimated  by  observing  the  feature  from  two  different  locations  of 


the  platform  and  using  inertial  displacement  and  orientation  change  measurements.  This 
estimation  is  derived  from  Equation  (8)  and  is  formulated  as  follows: 


P«=(h;(.Hp)^'.H 


N  'B'^RiNS 


-C 


H„ 


(eff.AC 


B*  -ACb  -e, 


N  ,h(2) 

P 


•  e 


(1) 


(10) 


Essentially,  the  image  depth  is  initialized  using  synthetic  stereo-vision:  camera  motion  is  applied 
to  synthesize  a  stereo-vision  baseline,  which  is  measured  by  the  INS.  Note  that  in  this  case  a 
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correlation  between  depth  estimation  errors  and  inertial  position  and  orientation  errors  is 
introduced.  This  correlation  must  be  taken  into  account  by  the  Kalman  filter  design.  Particularly, 
the  filter  states  include  INS  error  states,  and  errors  in  initial  range  estimates.  Hence,  the 
correlation  between  INS  position  and  orientation  errors  and  range  estimation  errors  must  be 
incorporated  into  the  state  covariance  matrix  of  the  Kalman  filter.  An  approach  for  computing 
the  INS/vision  correlation  for  the  multi-aperture  case  is  similar  to  the  computation  of  INS/vision 
correlation  for  a  single  aperture  case,  which  is  described  in  [1]. 

Simulation  Results 

Performance  of  multi-aperture/vision  inertial  aiding  was  assessed  in  a  simulated  indoor 
environment.  Figure  9  shows  a  2D  projection  of  the  simulation  environment.  The  simulation 
scenario  included  motion  in  an  indoor  hallway.  Hallway  walls  were  simulated  as  vertical  with  a 
2.5-m  height.  For  each  wall,  four  point  features  were  uniformly  distributed  over  its  length  and 
height.  The  simulated  motion  trajectory  was  two-dimensional.  However,  the  vision/inertial 
system  estimated  all  six  degrees  of  motion  freedom.  Inertial  measurement  unit  was  simulated  as 
a  lower-grade  unit  with  an  accelerometer  bias  of  1  mg  and  a  gyro  drift  stability  of  50  deg/hr. 
Specifications  of  video  cameras  were  simulated  as  follows:  640x480  resolution,  10  Hz  update 
rate,  40  deg  by  30  deg  field-of-view  (FoV).  Errors  of  extracting  a  point  feature  from  camera 
images  were  simulated  as  a  camera  quantization  error  plus  a  random  Gaussian  noise  error  with  a 
standard  deviation  of  one  pixel. 
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Simulated  indoor  environment 


- walls 
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- motion  trajectory 
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Figure  9.  Simulated  indoor  environment  implemented  for  the  evaluation  of  multi-aperture  vision/INS 

integration 

Figure  10  exemplifies  simulation  results.  Position  aoeuracy  performance  is  shown  for  cases  of  a 
free-running  inertial,  a  single  camera/INS  integration,  and  a  multi-aperture  vision/INS 
integration  including  cases  of  three  and  eight  apertures.  For  the  multi-aperture  cases,  the  angular 
separation  between  individual  cameras  was  implemented  as  120  deg  and  45  deg  for  the  three- 
aperture  and  eight- aperture  cases,  accordingly. 

Simulation  results  presented  demonstrate  that  for  the  simulation  scenario  implemented  the  use  of 
multi-aperture  INS  aiding  allows  reducing  position  errors  by  two  orders  of  magnitude  as 
compared  to  the  single-aperture  case:  position  drift  is  reduced  from  a  100-m  level  to  a  1-m  level. 
The  accuracy  performance  is  improved  due  to  the  following  factors.  Firstly,  the  number  of 
available  features  generally  increases  by  increasing  the  camera  FoV  using  multiple  apertures. 
Secondly,  and  more  importantly,  the  use  of  multiple  apertures  enhances  the  feature  range 
initialization  capabilities.  Figure  11  illustrates  this  effect. 


14 


Free-running  INS:  position  errors,  km 


Single  aperture/INS:  position  errors,  m 


Three  apertures/INS:  position  errors,  m 
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Eight  apertures/INS:  position  errors,  m 
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100  time  s-oo 

^  time,  s 

Figure  10.  Example  simulation  results  for  the  indoor  simulation  environment 


Large  angular  change  is 
accumulated  for  the  feature 
observed  by  the  side-looking  camera 
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Small  angular  change  is 
accumulated  for  the  feature  observed 
by  the  forward-looking  camera 


Figure  10.  Improved  range  initialization  capabilities  using  multi-aperture  vision:  large  angular  change  is 
accumulated  over  a  small  distance  for  the  feature  observed  by  the  side-looking  camera;  this  feature  is  initialized 
and  then  applied  to  correct  inertial  drift  over  a  longer  distance  that  is  required  to  accumulate  an  angular  change 
sufficient  for  initializing  the  feature  observed  by  the  front-looking  camera 
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The  range  estimation  accuracy  is  primarily  influenced  by  the  feature  angular  change  between 
images  that  are  used  for  the  estimation.  The  determinant  of  the  hJ  •  Hp  matrix  in  the  pseudo¬ 
inverse  formulation  of  equation  (10)  is  directly  related  to  changes  in  spherical  angles.  An 
increased  angular  change  increases  the  determinant  value  and  thus  improves  the  estimation 
accuracy.  The  maximum  angular  change  is  accumulated  when  passing  over  a  feature  and 
observing  this  feature  with  a  side-looking  camera  as  shown  in  Figure  11.  As  a  result,  this  feature 
is  efficiently  initialized  after  a  small  platform  displacement.  Kalman  filter  observables  for  this 
feature  are  then  applied  to  correct  inertial  drift  over  longer  displacement  intervals  that  are 
required  in  order  to  accumulate  an  angular  change  that  is  sufficient  to  initialize  the  feature  range 
for  the  front-looking  camera. 

To  illustrate  the  efficacy  of  using  the  multi-aperture  vision  approach  for  the  range  initialization, 
position  accuracy  of  the  multi- aperture/INS  system  was  compared  to  performance  of  a  stereo¬ 
vision-aided  inertial.  For  the  stereo  case,  a  two-camera  system  was  implemented.  Images  of  both 
cameras  are  converted  into  a  unit  sphere  frame  and  process  of  that  frame.  The  stereo-vision 
principle  is  applied  to  initialize  unknown  ranges  of  video  features.  Performance  of  the  stereo¬ 
vision  system  with  a  1-m  baseline  was  accessed  in  the  simulated  indoor  environment  shown  in 
Figure  9. 

Figure  12  shows  position  performance  of  the  stereo-vision-aided  inertial.  In  this  case,  position 
errors  stay  at  a  meter-level  and  the  maximum  error  in  horizontal  position  components  is  4  m, 
approximately. 
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Figure  11.  Simulation  results  for  the  stereo-vision/INS  integrated  case 

Simulation  results  shown  in  Figures  10  and  12  demonstrate  that  similar  to  the  multi-aperture 
case,  the  use  of  stereo-vision  allows  for  meter-level  positioning  accuracy.  It  is  noted  that  multi¬ 
aperture  vision  still  delivers  a  better  performance:  maximum  position  error  is  about  2  m  for 
multi-aperture  case  and  about  4  m  for  the  stereo-vision  case.  Hence,  it  can  be  inferred  that  the 
use  of  multi-aperture  imagery  allows  for  range  initialization  capabilities  that  are  at  the  same  level 
or  better  as  compared  to  stereo- vision  systems  with  long  baselines.  The  main  advantage  of  the 
multi-aperture  approach  is  its  scalability  and  direct  applicability  for  small  platform  application 
scenarios  where  large  baseline  cannot  be  utilized. 

Overall,  simulation  results  presented  in  this  section  demonstrate  the  efficiency  of  the  multi¬ 
aperture  vision/INS  data  fusion  methodology  that  is  developed  in  this  paper. 
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Conclusion 

This  paper  proposes  the  use  of  multi- aperture  imagery  integrated  with  inertial  measurements  for 
navigation  in  GNSS-denied  environments.  A  unit  sphere  representation  of  features  that  are 
extracted  multi-aperture  images  was  developed.  Motion  constrains  that  relate  feature  parameters 
to  the  motion  of  cameras  were  represented  in  the  unit  sphere  frame.  The  unit  sphere  motion 
constraints  were  then  applied  to  formulate  multi-aperture  vision/INS  data  fusion.  Performance  of 
the  integrated  solution  was  accessed  in  the  simulated  indoor  environment. 

Future  efforts  will  evaluate  performance  characteristics  of  the  multi-aperture/inertial  integration 
using  experimental  data  including  urban  ground  data  and  flight  test  data. 
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